Papers by João Maria Janeiro
MEXMA: Token-level objectives improve sentence representations (2025.acl-long)
Copied to clipboard
| Challenge: | Current approaches to cross-lingual sentence encoders use sentence-level objectives only. |
| Approach: | They propose a novel approach that integrates both sentence-level and token-level objectives. |
| Outcome: | The proposed approach outperforms existing CLSEs on bitext mining tasks and downstream tasks. |
Mixture of Languages: Improved Multilingual Encoders Through Language Grouping (2025.emnlp-main)
Copied to clipboard
João Maria Janeiro, Belen Alastruey, Francisco Massa, Maha Elbayad, Benjamin Piwowarski, Patrick Gallinari, Loic Barrault
| Challenge: | Recent work in this field relies on training transformer encoders on a large amount of multilingual data, with all parameters shared across all languages. |
| Approach: | They propose a mixture of languages strategy to pretrain largely multilingual encoders using masked language modeling. |
| Outcome: | The proposed architecture outperforms a dense counterpart, MoE models and public multilingual encoders on downstream tasks while minimizing interference. |